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(57) Abstract: In a particular embodiment, the disclosure is directed to a method of recognizing input that includes receiving input 
^ data; receiving context data associated with the input data, the context data associated with an interpretation mapping; and generating 
O symbolic data fmm the input data using the interpretation mapping. In another particular embodiment, the disclosure is directed to 
^ an input recognition system that includes a context module, an input capture module, and a recognition module. The context module 

is cunflguned lo receive context input and provide context data. The input capture module is conflgured to receive input data and 
^ is configured to provide digitized input data. The recognition module is coupled to the context module and is coupled to the input 

capture module. The recognition module is configured to receive the digitized input data and to interpret the digitized input data 
^ utilizing an interpretation mapping associated with the context data. 
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METHOD AND SYSTEM FOR CONTEXT-SENSITIVE RECOGNITION OF 

HUMAN INPUT 

Randolph Lipscher 
Michael D. Dahlin 

CROSS-REFERENCE TO RELATED APPLICATIONfS) 

11001] The present application claims priority from U.S. provisional patent application 
no. 60/402,498, filed August 8, 2002, entitled "Method and Apparatus for context-sensitive 
recognition of human input," naming inventors Randolph B. Lipscher and Michael D. 
Dahlin, which application is incorporated by reference herein in its entirety. 

FIELD OF THE INVENTION 

[1002] This invention generally relates to human input recognition. More specifically, 
this invention relates to voice and handwriting recognition using context-sensitive . 
recognition and human-assisted feedback correction. 

BACKGROUND 

[1003] Various human inputs into computation systems require interpretation by 
computer. E?camples include Voice and handwriting recognitioiL Typical interpretation 
systems require intense computation or rely on single broad dictionaries to interpret die 
input As such tibiese systems are slow and unreliable. The lack of speed often leads to the 
systems fidling behind, leaving gaps in die output fiom (he interpretation. 

[10041 Furllier, these systems are prone to error. The error is in part caused by speed of 
the system relative to real-time human input speeds. In addition, error is caused by 
misinterpretation of the input 

SUMMARY 

(10051 In a particular embodiment, the disclosure is directed to a method of recognizing 
input. The method includes receiving input data; receiving context data associated with the 
input data, the context data associated with an interpretation mapping; and generating 
symbolic data from the input data using the interpretation mapping. 

f 10061 In another particular embodiment, the disclosure is directed to an input 
recognition system. The input recognition system includes a context module, an input 
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capture module, and a recognition module. The context module is configured to receive 
context input and provide context data. The input capture module is configured to receive 
input data and is configured to provide digitized input data. The recognition module is 
coupled to the context module and is coupled to the input capture module. The recognition 
module is configured to receive the digitized input data. The recognition module is 
configured to interpret the digitized input data utilizing an interpretation mapping associated 
with the context data. 

(1007] In another particular embodiment, the disclosure is directed to a medical system. 
The medical system includes at least one input capture module, a context module, a plurality 
of interpretation mappings, and a recognition module. The at least one input capture 
module is configured to capture iiqiut data and provide digitized input data. The context 
module is configured to receive medical workflow data and provide cont^ data. The 
context data is associated with at least one interpretation mapping of the plurality of 
interpretation mappings. The recognition module is configured to generate symbolic data 
from the di^tized input data utiliang the at least one mapping associated with the context 
data. 
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BRIEF DESCRIPTrON OF THE DRAWINGS 

(10081 FIG- 1 illustrates an embodiment of a natural input recognition system. 

(10091 FIG- 2 depicts an exemplary method of input recognition. 

[10101 FIG. 3 illustrates an exemplary embodiment of a natural input recognition 
system. 

[10111 FIG- 4 depicts an exemplary mefliod for input recognition. 

[10121 FIG. 5 illustrates an exemplary embodiment of a natural input recognition 
system. 

[10131 FIG. 6 depicts an exemplary method for input recognition. 

[10141 FIG. 7 illustrates an exemplary embodimmt of a natural iiq>ut recognition 
system. 

[10151 FIG. 8 depicts an exemplary method for input recognition training. 

[10161 FIG. 9 illustrates an exemplary embodiment of a natural input recognition 
system. 

(10171 FIG. 10 depicts an exemplary mefliod for input recognition training. 

(10181 FIG. 11 illustrates an exemplary embodiment of a natural input recognition 
system. ' 

(10191 FIG. 12 depicts an exemplary mefliod for input recognition training. 

(10201 FIG. 13 illustrates an exemplary embodiment of a natural input recognition 
system. 

110211 FIG. 14 depicts an exemplary method for input recognition. 

(10221 FIG. 15 illustrates an exemplary embodiment of an input capture module. 

110231 FIG. 16 illustrates an exemplary embodiment of a feedback module. 
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(10241 FIG. 17 illustrates an exemplary embodiment of a recognizer module. 

[10251 FIGs. 18, 19, 20, and 21 illustrates an exemplary embodiment of a natural input 
recognition system. 

(10261 FIG. 22 depicts an exemplaiy embodiment of a context module. 

[10271 FIG. 23 depicts an exemplary application of context-sensitive recognition. 

(10281 The use of the same reference symbols in different drawings indicates similar or 
identical items. 
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DESCRIPTION OF THE DRAWINGS 

11029] This disclosure describes a natural human input recognition system that is 
applicable to recognition systems such as voice-to-text translation or handwriting-to-text 
translation. 

[1030] FIG. I illustrates an embodiment of a natural input recognition system. Natural 
input 102 is directed to a recognition system 104. The recognition system 104 generates 
symbolic data ficom the natural input 102. 

{1031] A human input recognition system takes natural input as input and produces 
symbolic data output Natural input may be any form of input produced by a human or 
communication form suitable for human-to-human communication. Examples may include 
voice, speech, gestures, handwriting, fecial expression, or a drawing/sketch/schematic. 
Symbolic data are collections of values that can represent data in a computer. Examples may 
include words, phrases, letters, numbers, Unicode symbols, values for database record, 
computer program variable values, and computer program variable addresses. Symbolic . 
data output may be output by tiiie system, stored by the system, displayed by &e system, or 
transmitted to another system. However, the symbolic data and natural iiq>ut may take 
various forms. Furtiher, various conversions may be envisaged. 

11032] FIG. 2 is a flow chart describing the actions taken by an embodiment of a natural 
input recognition systeoL A user provides natural input to the system as shown in step 202, 
and the system produces symbolic data corresponding to that natural input, as shown in step 
204. 

[1033] FIG. 3 illustrates an embodiment of the natural input recognition system that also 
takes context as input. In one embodiment, Oie system 306 takes natural input 302 and 
context input 304 and produces symbolic data 308. The system 306 adapts the 
interpretation of die natural input 302 based on the context input 304. The system 306 may 
utilized a specialized mapping based on the context input 304. Alternately, the system 306 
may select a set of interpretation mapping based on the context input 304. 

[1034] Context is information describing the situation in which the input is provided. 
Examples of context include the task being performed such as administering a medical 
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physical exam, writing a medical prescription, administering a medical physical exam of the 
hand, administering a medical physical exam for someone who has complained of back 
pain, ordering a blood test for a medical patient, tuning an automobile engine, repairing an 
automobile engine for a 1997 Ford Mustang with a V-8 engine, repairing an automobile 
engine for a 1997 Ford Mustang with a V-8 engine that makes a clicking sound, taking class 
notes about calculus, taking class notes about chapter 5 of the Calculus textbook Calculus 
with Analytic Geometry Second Edition by Howard Anton, entering sales data, entering 
sales data about auto parts, and entering sales data about manual transmission auto parts for 
Ford vehicles, among others. 

(1035] The context may include a single data context such as writing a prescription. 
Alternately, tiiie context may include a set of hierarchical data. For example, a physical 
exam of flie hand may include physical exam context information and hand context 
information. 

(10361 Another example of context information is the ^pe of subject being examined. 
For example, in a medical application a patient! s demographic information - such as age, 
gender, race, income, and location of residence — could act as context information. For 
example, in an auto repair application, factors such as a car's make, model, trim level, and 
year of manufacture could act as context information. For example, in a sales q)plication, 
fectors such as customer's type of business or number of employees could act as context 
mfonnation. 

[10371 A further example of context information is stored information about the subject 
of an examination or procedure. For example, in a medical application, information stored 
about a patient being medically examined such as the patient's age, gender, name, past 
medical history findings, current and past medications, recent diagnoses, chief complaint, 
history of present illness IHndings, and so on could serve as context information. 

110381 For example, in an auto repair application, information such as past repairs, 
recently diagnosed problems, and so on could act as context information. For example, in a 
sales application, information such as item numbers in past sales to a customer, descriptions 
of items in past sales to a customer, recent correspondence with a customer, and so on could 
act as context information. 
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[10391 Another example of context information is the current or recent physical location 
of the user. For example, in a real estate application, a real estate agent dictating to a laptop 
that includes a GPS could use the location of the agent as context. For example, in a 
medical application, the room that a health care provider is in or was last in could be 
regarded as context information. 

[1040] Context information may also include the subroutine of a computer-aided 
workflow. For example, if a workflow has several steps that take natural input, then the step 
currently in progress could act as context in the recognition syst^. For example, in a 
voice-driven telephone customer service application, one example context could be the 
"confirm customer address" task while another example context could be flie "receive 
ordered item number" task. For example, in a graphical computer input interface 
application, the window or frame that the user last touched with a mouse click or a stylus 
tap could represent tiiie current context 

[1041] In a multi-component context embodiment, one or more types or items of 
information may be combined to represent a multi-component context For exan^le^ in one 
embodiment of a medical point-of-caie electronic medical lecord application, several factors 
such as the current patient (e,g., Mr. Jones, age 55, male), the chief complaint (e.g., chest 
pain), the diagnosis entered during this encounter (e.g., heartburn), and the current task (e.g., 
write prescription) could together represent the context 

[1042] Different contexts can be active at different times for the same user. A context 
change might not directly update mappings between a particular natural input and the 
corresponding symbolic data recognized by the system. Instead, it may change a collection 
of one or more mappings. For example, selecting the context "fruit" rather than the context 
"general" might not directly alter the mappings from natural inputs to either the words 
"fruit" or "general" while it might alter the mappings from the space natural inputs to oflier 
words, for example increasing the probability that given inputs map to the words "orange," 
"lime," and "grape" while reducing the probability that the given inputs map to the words 
"porridge," "time," and "great." 

[10431 FIG. 4 is a flow chart describing the actions taken by an embodiment of a natural 
input recognition system that accepts context input. In this embodiment, the system 
receives context input, as shown in step 402. The user provides natural input to the system. 
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as shown in step 404. and then, the system produces symbolic data correspondmg to the 
natural input in the specified context, as shown in step 406. The user may continue to 
provide additional natural input in this context, and the system will produce additional 
symbolic data by interpreting the natural input in the current context. Alternatively, a new 
context may become active, at which point future natural input will be interpreted in the 
new context Notice that the same natural input may produce different symbolic data 
outputs if that natural input is provided different contexts. For example, in a handwriting 
translation for electronic medical record embodiment, the same natural input might be 
interpreted as "Mrs. Johnson" when the context is that flie current patient is a female named 
Qaire Johnson and as "Mrs. Johnstone" when flie context is that current patient is a female 
named Amy Johnstone. 

(10441 FIG- 5 illustrates an embodiment that also takes context change as input. In this 
embodiment, the system 508 takes natural input 502, context 504, and context change 506 
as input and produces symbolic data 510. Context change is any alteration of the relevant 
context data that affects the mappmg of natural mput to symbolic data. Two sample types 
of context change are navigation and context update. 

110451 Navigation inpats are inputs that change what set of information is relevant 
context For example, navigation mputs may include selecting a computer menu item, 
selecting a graphical window, selecting a graphical window fiame, selecting a task, 
completing a task, selecting a patient, selecting a subject, or entering information, findings, 
or orders about a patient or subject In one embodiment, navigation inputs are supplied as 
digital or discrete input, such as selecting an item by a mouse click, stylus tap on a touch 
screen, or fmger tap on a touch screen. In another embodiment, navigation inputs are 
supplied as natural input, such as saying the words "next screen", saying the name of a task, 
providing natural input that completes a task, making a gesture in the air with a hand, 
shaking or nodding one's head, or shaking Ae input device in the air to activate a motion 
sensor. 

[10461 Context update input is any input that adds, modifies, or deletes elements from 
the current context For example, in a medical context, the "History of Present Illness" 
context might include information relating to findings about the current patient that have 
been entered into the system. As new findings are entered, an embodiment of the system 
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updates the context to include these new findings and infonnation relating to these findings 
in the context 

[10471 FIG. 6 is a flow chart describing the actions taken by an embodiment of a natural 
input recognition system that accepts context change input. As shown in step 602, the 
system receives context change input The system then changes, selects, or updates a 
context based on this navigation input, as shown at step 604. The system then receives 
natural iiq>ut, as shown at step 606, and using the context and the natural input, flie system 
produces symbolic data corresponding to the natural input interpreted in the current context, 
as shown at step 608. The user may continue to provide natural input by repeating step 606, 
or the user may provide navigation input by repeating step 602. 

[10481 FIG. 7 illustrates an embodiment in which the system uses feedback ftom usere 
to adjust the algorithms or training data used internally by its recognition system. Tlie 
system 708 produces symbolic data 710 from the natural input 702 utilizing training data 
706. The training data is derived at least in part from feedback 704. 

[1049] Training data is data that encodes patterns of natural input to symbolic data 
mappings for a user or group of users. For example, statistical information about the words 
or phrases that a user conmionly users is one type of training data. For example, statistical 
information about a user's speech patterns and tiie resulting symbolic data (words) is one 
type of training data. Methods for using training data to enhance natural iiq)ut recognition 
include calculating conditional probabilities, conjGguring neural networks, decision trees, 
and the like. 

[10501 It should be noted that context (described above) dififera fipom training data. For 
one thing, context can represent activities, subjects, topics of information, while training 
data represents mappings from natural input to symbolic output independent of context In 
one embodiment, training data is associated with a user or group of usera while context is 
associated with a task or subject A set of training data may be selected from a library of 
training data based on the context data. 

(10511 FIG. 8 is a flow chart describing the actions taken by an embodiment of a natural 
input recognition system that utilizes feedback for training. In this embodiment, the system 
receives natural input, as shown in step 802, and generates symbolic data, as shown in step 
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804. The system may continue to receive natural input and generate data, or at any point, 
the system may receive feedback, as shown at step 806, which it uses to update its training 
data to unprove future recognition. For example, in one voice recognition embodiment, 
after a user says "apple," the recognition system might produce the symbolic data "attle." 
The user would recognize the error on the screen, select the word "atlle" on the screen, and 
activate a correction subroutine by typing the word "apple." The system would then update 
its data, as shown in step 808, to increase the probability that when the user makes sounds 
similar to the sounds she just made, the system will be more likely to recognize fliose 
sounds as the word "apple" and less likely to map those sounds to the word "atde." 

(1052) FIG. 9 illustrates an embodiment that combines feedback and context In this 
embodiment, feedback 908 is used to update mappings from particular sets of natural inputs 
902 to sets of symbolic data 912, and context 904 is used to adjust or select collections of 
such mappings. For example, the feedback subsystem would update the probability of 
recognizing a collection of sounds as tihe word "apple" raflier than "addle" when flie user 
corrects a mistranslation of a spoken word. For example, tihie context subsystem would 
update the probability of recognizing a collection of sounds as tiie word "apple" when flie 
user selects the "shopping for fruit" context as opposed to the "general context" or the 
"shopping for electronic equipment" context 

[1053] In one embodiment, feedback updates natural input to symbolic output mappings 
for die current context In one embodiment, feedback updates global m^pings diat are 
relevant to all contexts. In one embodiment, feedback updates both per-context mappmgs 
and global mappings, witti differing weights on the updates. 

[1054] FIG. 10 is a flow chart describing the actions taken by an embodiment of a 
natural input recognition system that utilizes feedback and context In this embodiment, the 
system receives context change input and context input, as shown in steps 1002 and 1004. 
The system receives natural input, as shown in step 1006, and generates symbolic data, as 
shown in step 1008. It may continue to receive natural input or context input and repeat 
these actions. Or it may receive feedback, as shown in step 1010, which it uses to update its 
training data, as shown in step 1012. 

[10551 FIG- 1 1 illustrates an embodiment in which two users 1 102 and 1 106 interact 
with the system. The first user 1 102 provides natural input 1 104 and the system 1 1 10 
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^nerates symbolic data 1 1 12. The system then transmits the symbolic data 1 1 12 to the 
second user 1 106. The second user 1006 provides feedback 1 108 (e.g., corrections to the 
symbolic data), which the system 1110 then uses to improve its recognition mappings. 

(10561 In a per-user training data embodiment, flie updates provided by flie second user 
1 106 update the training data that the system 1110 uses for recognizing natural input by the 
first user. 

[10571 In one embodiment, both the symbolic data 1 1 12 and the natural input 1 104 are 
sent by the system to the second user 11 06. The second user 1 106 then has access both to 
the original natural input 1 104 and the generated symbolic output 1112 when providing 
feedback 1108. 

[1058] For example, in a speech recognition dictation embodiment: user A speaks, 
system displays proposed symbolic data to user B (while, optionally, playing flie original 
speech fluou^ speakers or headphone to user B), user B selects/corrects symbolic data; 
corrected words go back to recognition systen^ recognitions system marks the selected 
words as "morel likely" and/or adds any new words to its internal symbolic dictionary. 

[1059] In one embodiment the system stores the natural input and the symbolic data 
before sending it to the second user. The second user ttius may provide feedback "oflFlme" - 
- at a time considerably after the first user provides the natural input In one embodiment, 
the system stores the natural input and does not immediately generate symbolic data. The 
symbolic data is generated at a later time. The second user ttien provides feedback. 

[1060] FIG. 1 2 is a flow chart describing the actions taken by an embodiment of a 
natural input recognition system in which two users interact with the system. The first user 
provides natural input, as shown in step 1202, and the system generates symbolic data, as 
shown in step 1204. The system then transmits the symbolic data to the second user. The 
second user provides feedback, as shown in step 1206 (e.g., corrections to the symbolic 
data), which the system then uses to update its training data, as shown in step 1208. 

[10611 FIG. 13 illustrates the main modules of an embodiment of a recognition system. 
In this embodiment, a context module 1306 generates the appropriate context 1308 and 
feeds it to the recognizer module 1316. The context module 1306 accepts context input 
information 1302 (i.e., the context to use is provided fi^om an external source) or context 
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change information 1304 (i.e., the context module maintains context state that is updated) or 
both. As noted above, in one embodiment content change information 1304 can be 
navigation infomation or content update information or both. The context input 1302 and 
context change information 1304 can be supplied from various types of sources such as 
from external sources (such as other computers, other programs, or computer networks), 
from digital user input (such as selecting a menu item, making a window active, checking a 
checkbox), or from symbolic output from the recognizer (such as words to store or 
navigation commands). 

(1062] In this embodiment, flie input capture module 1312 captures human natural input 
1310 (such as voice, gestures, handwriting, sketches) and produces a digital natural data 
encoding 1314 (such as a stream of bits on a wire, an array of bytes on a network, or typed 
data in a computer program). 

[1063] The recognizer module 1316 produces symbolic data 1318 based on digital 
natural data 1314, context data 1308, and feedback data 1324. 

[1064] The feedback module 1320 receives digital natural input 1314, symbolic 
datal318, and user feedback 1322 and produces feedbackl324. In one embodiment, this 
feedback 1324 represents (he intended symbolic data that should have been produced for the 
specified digital natural iiq>ut 1314. 

[1065] Tliese modules may run together on a singje system, or separately on various 
systems, or in various combinations. However, various system configurations may be 
envisaged. For example all of the system elements may be run on a computer, collection of 
computers, and networks, among ottiers, with various storage, memory, and processors, 
among others. 

(10661 While the diagram illustrates direct flows of data between modules, that these 
data flows may be accomplished via a number of means such as computer DRAM memory, 
computer non- volatile disk storage, computer networks, procedure calls, remote procedure 
calls, asynchronous messaging such as IBM*s MQS system, and combinations of means. 
Not all data flows need to use the same communication means. It will further be apparent 
that in some embodiments, one or more of these communication flows may be 
asynchronous, in which case considerable time may elapse between the production of data 
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by one module and its consumption by another. For example, in one embodiment digital 
natural input and context data may be stored on disk for several hours before being fed to 
the recognizer system. Furthemore, in some embodiments, different users can provide 
different subsets of the inputs. For example, in one embodiment, one user may provide the 
natural input while another provides feedback. 

f 1067] FIG. 14 is a flow chart describing the actions taken by an embodiment of a 
natural input recognition system. In this embodiment, as shown in step 1402, the context 
module receives context input or context change data, as shown in step 1404, generates the 
relevant context, and, as shown in step 1406, sends it to the recognizer module. If fee next 
iiq)ut is context input or context change data, the system returns to step 1402. 

[10681 Otherwise, if the next input is natural input, as shown in step 1406, the input 
capture module receives natural input, as shown in step 1408, digitizes it, and as shown m 
step 1410, sends it to Ae recognizer module. As shown in step 1414 the recognizer module 
then produces symbolic data. As shown in step 1416, flie recognition module sends the 
symbolic data to flie feedback module, which receives it as shown in step 141 8. Then, if the 
next input is cont^t input or context change data, the system returns to step 1402. 

11069] Otherwise, if the next input is natural input, the system returns to step 1406. 
Otherwise, if the next input is user feedback, the system proceeds as shown in step 1420, in 
which the feedback module receives feedback input Then, as shown in step 1422, die 
feedback module sends feedback to the recognizer. As shown in step 1424, the recognizer 
receives the feedback. Then, as shown in step 1426, the recognizer updates the mapping 
from digital natural inputs to symbolic data according to fliis feedback. Depending on the 
next input, the system then proceeds to step 1 or step 4. 

110701 FIG. 15 illustrates an embodiment of an input capture module. In this 
embodiment, the input capture module 1 504 captures human natural input 1 502 (such as 
voice, gestures, handwriting, sketches) and produces a digital natural data encoding 1506 
(such as a stream of bits on a wire, an array of bytes on a network, or typed data in a 
computer program). A large number of such systems will be familiar to designers familiar 
with flie art. Examples include analog microphones with analog-to-digital conversion 
boards such as are found with many commodity SoundBlaster (TM) compatible audio cards, 
microphones with USB digital connections, touch screens and styluses such as available on 
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the Palm, Inc. Palm Vx (TM) computer and on the tablet fomi-factor Hitachi HPW-600ET 
computer, and such as a distal video camera such as the Oregon Scientific Inc Y-Cam, 
which captures video and produces digital data with a USB interface. 

[10711 In &e exemplary embodiment shown in FIG. 16, flie feedback module 1608 
receives digital natural input 1602, symbolic data 1604, and user feedback 1606 and 
produces feedback 1610. In one embodiment, this feedback 1610 represents the intended 
symbolic data ttiat should have been produced for the specified digital natural input. In one 
embodiment, flie feedback 1610 is simply encoded as the symbolic output that should have 
been produced by flie recognizer for the last digital natural vapvA received by the recognizer. 
In a second embodiment, each set of symboUc data sent by the recognizer to the feedback 
module 1608 includes a unique identifier, and the feedback 1610 sent fiwm the feedback 
module 1608 to lite recognizer is encoded as flie unique identifier or identifiers for flie 
symbol or symbols to be correcled followed by flie symbolic data fliat should be substituted 
for flw symboUc data 1604 original^ produced. Such an embodiment would be appropriate 
for allowing flie feedback module to correct a range of characters in an ASCH or Unicode 
textbu£fer 

(1072J In one embodiment, flie feedback module 1608 does not rely on digital natural 
input, and thus, iiqiut may be omitted firom flie module. One example of such an 
embodiment is a digital speech to text system in which flie feedback module 1 608 displays 
flie generated symbols (ie., text) and allows correction of fliis text using keyboard or mouse 
driven text-editing commands. In anoflier embodfanent, flie feedback module 1608 emits 
bofli flie natural mput and flie qmbolic output to fecilitate feedback. For example, in a 2- 
person dictation embodiment, a first person dictates text verbally, and a second person 
receives bofli flie system generated symbolic text and a distal recording of flie original 
dictation sounds. The second person bofli listens to flie sounds and looks at tfie jwoduced 
text in order to identify errors and provide feedback. 

110731 FIG- 17 illustrates the inputs and ou^uts of an embodiment of the recognizer 
subsystem. The recognizer subsystem 1708 takes as input digital natural input 1702 and 
produces symboUc data 1710 as ou^ut In one embodiment, it also takes context 1704 as 
input. Different contexts may cause tiie same digital natural input to be mterpreted in 
different ways - e.g., to produce different symbolic data outputs. In one embodiment, it 
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also takes feedback 1706 as input Feedback 1706 specifies the correct translation fipom a 
specific digital natural input set to a specific symbolic data set. 

[1074] FIG. 18 illustrates an embodiment in which context is used to select from among 
the outputs of multiple recognizer algorithms. In diis embodiment, digital natural input 
1804 is sent to several diffeient specialized recognizers (1810, 1812, 1814, and 1816) or a 
general recognizer 1818. The context 1802 may be used in conjunction with a router to 
route the digital natural input 1804 to the recognizeis (1810, 1812, 1814, 1816, and 1818). 
Each of the specialized recognizers (1810, 1812, 1814, and 1816) is designed and tuned to 
work well for a particular subset of contexts. In one embodiment, each specialized 
recognizer (1810, 1812, 1814, and 1816) is a complete natural-input-to-symbolic data 
system. Each copy of the system has been tuned to work well in a particular context - for 
example, by instantiating it with a different dictionary or language model of words and 
phrases and their probabilities of use. 

(10751 Alternately, the context input may be fed to a multiplexor (MUX) 1820, which 
selects the symbolic data output bom one of the recognizers (1810, 1812, 1814, 1816, and 
1818) according to the context 1802. 

[1076] In addition, if feedback 1 806 is supplied that indicates that natural input X 
should correspond to symbolic data Y, the router ensures that the feedback 1 808 is routed to 
only the specialized recognizer that corresponds to the current context 

(10771 For example, in one medical data input embodiment, four contexts are numbered 
0 C'general medical"), I ("prescription pad"), 2 ("history of present ilhiess"), and 3 ("enter 
diagnosis"), and flie context supplied corresponds to flie current phase of the medical 
encounter or task being performed by the physician using the system. In one embodiment, 
each specialized recognizer produces its best selection of symbolic data corresponding to 
each natural input, but only the set of symbolic data relevant to the current context is 
emitted by the system. In another embodiment, the digital natural iiq)ut is directed to a 
selected specialized recognizer, resulting the symbolic output 1822. 

(10781 In one embodiment, rather than always selecting the output from the relevant 
context as in the multiplexor embodiment illustrated above, the system weights different 
outputs more heavily depending on the context For example, in an embodiment, each 
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specified symbolic output is a correct translation of the digital natural input In this 
embodiment, the context selects a weighting of the specialized recogijitions. For example, 
in a variation of the medical input embodiment described above, when context 3 ("enter 
diagnosis") is active, the weights to different predictions are set to (0.5, 0.0, 1.0, 0.0), 
meaning that the "general medicine" prediction will be selected if its specialized predictor's 
confidence in its prediction is twice as high as flie "enter diagnosis" prediction (and the 
predictions of the "prescription pad" and "history of present illness" specialized predictors 
are ignored.) 

I1079I FIG. 19 aiustrates an embodiment of the recognizer in which different contexts 
use the same basic recognizer subsystem but make different data sets active. In one 
embodiment, instead of each specialized recognizer being a complete natural-input-to- 
symbolic-output subsystem, all conceptual specialized recognizers are in fact implemented 
by the same recognizer algorithm subsystem. This subsystem is parameterized in order to 
work well in different situations. As illustrated in FIG. 19, the context 1910 is used to select 
which parameters and state are available to fee recognition subsystem by selecting datal 
(1902), data2 (1904), data3 (1906), or data4 (1908) to be accessed by the recognizer 
algoriflun 1912. Each of the different data sets (1902, 1904, 1906, and 1908) comprises one 
or more collections of input to the recognizer algorithm 1912 such as a dictionary of words, 
a set of (word, probability) pairs, a set of phrases, a set of (phrase, probability) pairs, or a set 
of (natural input, phrase, probability) tuples. Also in fliis embodiment, feedback 1917 that 
updates flie mapping ftom natural input to symbolic data is used to update the active data 
set. The recognizer algorithm 1912 converts the digital natural input 1914 to symbolic data 
1918 using the data set (1902, 1904, 1906, or 1908) associated with the context 1910. 

(1080] FIG, 20 illustrates an embodiment in which recognizer data is divided into user- 
dependent, context-dependent data and user-dependent, context-independent data. In 
another embodiment, the recognizer system breaks recognizer data into two parts. The first 
part contains data pertaining to user-dependent, context-independent data (UD/CI) 2002. 
The second part contains data pertaining to user-dependent, context-dependent data 
(UD/CD) (2008, 2010, and 2012.) For example, in one voice to text embodiment, user- 
dependent, context-independent data (2002) comprises data describing a user's 
pronunciation of different words while user-dependent, context-dependent data comprises 
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data about the frequency with which different words and phrases are uttered in a cont«L In 
this embodiment, feedback is also split to update the corresponding subsets of data (2006 
and 2014). 

11081) In another embodiment, the recognizer data is also split into two parts wifli the 
same functional purposes. The first set is user-dependent, context-independent data 2002 but 
the second set is user-independent, context-dependent data (2008, 2010, and 2012) (i.e., data 
that corresponds to the context but that is collected across a collection of different users.) 

(10821 FIG. 21 illustrates an embodiment in which context-dependent data is supplied to 
the recognizer subsystem. The recognizer module 2106 utilizes that digital natural input 
2102 in conjunction with the context dependent data 2104 to produce the symbolic data 
2108. Raflier than storing context dependent data in tibie recognizer and selecting a set of 
context-dependent data using the context, the context-dependent data is provided directly as 
the context For example, in a medical handwriting recognition embodiment, the enclosing 
system provides a list of words relating to flie current patient (e.g., the patient's name, a list 
of the patientfs current medications, a list of past diagnoses that have been made about the 
patient, and a list of active problems for the patient) as well as a list of words relating to the 
current task. For example, one task is "history of present ilkess" (where, in this 
embodiment, words and phrases relating to the selected chief complaint are supplied; e.g., 
when flie chief complaint is chest pain and the current task is history of present iltoess, 
woids and phrases such as "chest", "heart", "smoking", "difficulty breathing", "fetigue", are 
supplied). In this embodhnent. oflier tasks are "write prescription", "enter diagnosis", "order 
laboratory test", "edit past medical, family and social history", "enter justification for MRI 
test", "comment on range of motion of right elbow", and so on. In one embodiment using 
this technique, the recognizer combmes context-dependent data with a context-independent 
"baseline" set of data. 

[10831 In a further embodiment, feedback 2110 applies to context independent training 
(e.g., updating models of the user's speech patterns) but feedback is used by the recognizer 
to update context-dependent data. 

(10841 FIG. 22 illustrates the basic input/output flows of one embodiment of the context 
module. The context module 2206 supplies context 2208 to flie recognizer module. The 
input 2202 to the context module is data that pertains to the situation in which Uie system is 



wo 2004/01S543 PCTAJS2003/02510S 

18 

being used In one embodiment, the context module 2206 maintains state regarding die 
current context, and context change bxputs 2204 alter that state. In another embodiment, the 
context module 2206 is stateless, and mformation encoding the current context is provided 
as input. In a Mrd embodiment, the context module 2206 maintains state regarding the 
current context, and this state is updated in two ways: incrementally (via context change 
inputs 2204) and en mass (via updates that encode the new context). 

[10851 In one embodiment, the context input 2202 can be considered to be of two types: 
(1) navigation input and (2) context update. These terms were defined above. 

[1086] The output of die context module 2206 is data that describes the current context 
In one embodiment, the output encodes the identity of a context 2208. For example, in one 
medical data input embodiment, four contexts are numbered 0 ("general medical"), 1 
("prescription pad"), 2 ("history of present illness"), and 3 ("enter diagnosis"), and the 
context output 2208 by die context module 2206 corresponds to the current phase of the 
medical encounter or task being performed by the physician using the system. In another 
embodiment, rather than naming the current context, the context module 2206 ou^uts 
context-dependent data such words or phrases that are relevant in the current context 

110871 In one multiple-contexts embodiment, multiple contexts are relevant at any given 
time, and flie context output of die contCTt module encodes tibiese multiple contexts. For 
example, in an embodiment where the context module outputs the identities of flie relevant 
contexts, a list of relevant contexts is output (e.g., "context = general, medical, HPI, chest 
pain, detail 'difBculty breafliing'"). For example, in an embodiment where die context 
module outputs per-context data such as words or phrases relevant to the cunent context, 
one multiple-contexts embodiment outputs flie union of die relevant words and phrases from 
the relevant contexts. 

[1088] One example type of multiple contexts embodiment is an embodnnent where 
different sets of contexts represent the situation along generally orthogonal sets of 
information. For example, in one medical embodiment, the current multiple-context 
includes three orthogonal factors: Uie current task, die current patient, and die current user's 
medical specialty. 
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[1089] Another example type of multiple contexts embodiment is an embodiment where 
different sets of contexts represent the situation along a hierarchical set of situations, where 
more specific subsets of context modify more general subsets of context. For example, in 
one medical embodiment, the current multiple-context includes up to three levels of 
hierarchical context - application area (e.g., "general medical", "financial", "personal"), 
application task (e.g.. "HPI", "ROS", "Diagnosis", "Prescription", "Order test", "Narrative"), 
and application sub-task (e.g., "comment on sore back", "write prescription for the 
medication penicillin", "Comment on MRI", and "Explain why an MRI is needed"). 

[10901 In a data entry template embodiment, a data entry template system comprises a 
number of screens and fiames. Each screen or ftame provides navigation means and a data 
input means. The navigation means makes anotiier screen or ftame active, causing the 
system to display the newly active screen or ftame. Ihe data input means provides means 
for entering data into the system. The data inputs means for each frame or screen comprises 
a digital data iiq)ut means (e.g., checkbox, radio button, selection list, keyboard text input 
box) or natural data input means (e.g., microphone for voice ii5)ut to the active ftame, 
screen for pen mput) or botit Data entered via data mput means is stored in the system. In 
one embodiment, the same input can be configured to activate bofli a navigation means and 
a data input means (e.g., selecting a radio button also changes a sub-frame on a screen). In 
this data entry template embodiment, natural input is duected to a particular screen or 
ftame, and this screen or ftame corresponds to tihte context in which the natural input is 
interpreted. In particular, flie context subsystem ou4>uts the context corresponding to flie 
currentiy active wmdow or ftame. In one embodiment, each window or frame's 
implementation comprises an XML file describing the window or frame. In tiiis 
embodiment, the XML file for a page or frame also comprises a list of words that are 
relevant context when the page or frame is active. 

[10911 In a medical field data entry template embodiment, the system comprises a 
number of screens and frames. The screens and fi»mes are arranged into a series of 
"applications", "tasks" and "sub-tasks." An exemplary navigation flow among tasks is 
illustrated in FIG. 23. In this flow, a user first logs m, as shown in step 2302, then selects an 
application (e.g., electronic medical record), as shown in step 2304. The user tiien selects a 
patient with which to work (e.g., from a list of patients in tiie clinic), as shown in step 2306. 
The user selects a task (e.g., HPI/ROS/Chief complaint 2308, Physical exam2310, diagnosis 
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2312. Rx 2316, oroflier tasks 2314). Hie user can then switch between tasks. Theusercan 
also flien navigate to a select patient screen to select a diflferent patient or the select 
application screen to select a different application (e.g., "check messages"), or finish the 
current patient and log out Furthermore, (not displayed in illustration) wittiin each task are 
several sub-tasks (e.g., within the HPI/ROS/Chief Complaint task are subtasks such as 
"comment on FINDING" where FINDING represents a data item that has been entered via 
digital input means and the subtask "comment on FINDING" provides the opportunity for 
the user to provide ftee-form natural input regarding the FINDING via handwriting 
recognition or voice recognition or both. In fliis embodiment, each task corresponds to a 
screen and each sub-task corresponds to a fiame within a screen. 

110921 The context module assembles the relevant context using bofli a hierarehical 
context and an orthogonal context means. In particular, the current context corresponds to 
the union of the contexts from (a) the current application, (b) the current patient (if any), (c) 
the current task within an application, and (d) the current subtask (if any). In one 
embodiment, each application, each task, and each sub-task is associated with an XML ffle 
that comprises infomiation to be displayed when the application/task/sub-task is active; the 
XML file also comprises a list of words and phrases that are likely to be entered when the 
application/task/sub-task is active. Furthermore, when a patient is selected, the system 
queries a storage system for records regarding that patient. The results of fliis query 
conH)rise a list of active problems, a list of allergies, and a list of current medications. Each 
element of these lists corresponds to one or more elements in a medical taxonomy or 
nomenclature such as the Center for Disease Control ICD9 code or the Medicomp Systems 
Medcin (R) nomenclature. Each element in the nomenclature is associated with zero or 
more relevant context words or phrases. The system takes the union of relevant context 
words or phrases fiom the findings associated with the current user, and the resulting set of 
words or phrases represent the patient-context. The system then takes the union of the 
patient-context and the application/task/sub-task contexts and this set represents the current 
context, which is output by the context module. 

[1093] In another embodiment, context relevant to the currently selected patient 
comprises one or more of the patient's name, words and phrases relating to ttie patient's past 
family medical and social history, words and phrases relating to the patients active or past 
problems, words and phrases relating to medications the patient has taken, words or phrases 
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relating to tests that have been performed on the patient, words or phrases relating to 
findings or orders entered into flie system regarding the patient during the cmrent medical 
encounter, and words and phrases relating to the patienfs demographics (e.g., gender, 
marital status, age). 

(1094] In another embodiment of a medical field data entry template system, rather than 
encode the context as a set of words and phrases, the context output by the system includes 
(a) the identity of the current application, task, and sub-task (if any) and (b) a set of words 
and phrases relevant to the current patient In this embodiment, the recognizer subsystem 
activates the specialized recognizers or recognizer state associated with the current 
application, tiie current task, and the current sub-task, and it also uses the words and phrases 
relevant to the current patient as input to its recognizer subsystems. 

[10951 In one medical field embodiment, each time a navigation action switches the 
active screen or fiame, the context output by the context module is updated. Furthermore, 
in this embodiment, each time a finding or other data is entered about the current patient, the 
context output by the context module is updated. 

[1096] In one medical embodiment, specialized context information is stored for 
different tasks such as HPI, ROS, PMFSH, orders, labs, Rx, enter diagnosis, coding, and 
narrative. Specialized context information may be stored for different categories of user 
such as for different roles (e.g., doctor, nurse, consultant, nurse practitioner, orderiy, 
paramedic, military field treatment) and such as for different specialties or clinic types (e.g., 
cardiologist, general practitioner, pediatrics, emergency room, geriatrics, military field 
treatment.) Specialized content information may be stored for different elements of 
information about a patient such as the patient's name, current/past medications, active 
problems, PMFSH, findings or data elements entered for the current encounter, and findings 
or data elements entered for past encounters. In one medical embodiment, specialized 
content information may be stored for different situations or patient populations such as flu 
season, responding to a mass casualty explosion, responding to an auto accident, responding 
to a poison gas attack, and so on. 

110971 The system described herein has application in a number of fields and systems. 
For example, in an auto mechanic embodiment, a template system provides data input and 
navigation means for various tasks on various types of automobile. Each screen or ftame in 
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the template system provides relevant context to the recognizer subsystem. Relevant 
context includes flie current task (e,g., changing oil, removing engine) and current subject 
(auto make, model and year). 

(10981 111 a student note-taking embodiment, the system uses the subject of the class ttiat 
the student is attending to select a class-specific vocabulary provided by the class's textbook 
publisher. This vocabulary acts as the relevant context during the class. The context 
module may also use the subject of the class that the student is attending to select a class- 
specific vocabulary provided by the class's textbook publisher. This vocabulary acts as the 
relevant context during the class. 

[10991 In a business note-taking embodiment, the system uses Bluetooth® to determine 
who else is in the room. Those names are relevant context The system may also use 
documents opened by user or previous notes with same people in room. These may be all 
context 

[11001 Th« recognition system may be use in various other applications such as delivery 
situations (e.g., UPS), automobile mechanics, students, medical applications, email dictation 
(other messages to/fi:om specified individual), shopping (standing in kitchen: using location 
sensor detect context; context is "in kitchen", predicting words that are used in kitchen), and 
retail sales. 

[11011 The above-disclosed subject matter is to be considered illustrative, and not 
restrictive, and the appended claims are intended to cover all such modifications, 
enhancements, and other embodiments, which fall within the true spirit and scope of the 
present invention. Thus, to flie maximum extent allowed by law, the scope of the present 
invention is to be determined by the broadest permissible interpretation of the following 
claims and their equivalents, and shall not be restricted or limited by the foregoing detailed 
description. 
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WHAT rS CLAIMED IS: 

1. A method of recognizing input, flie method comprising: 
receiving input data; 

receiving context data associated with the input data, the context data associated 

with an interpretation mapping; and 
generating symbolic data from die input data using the interpretation mapping. 

2. The method of claim 1, wherein the interpretation mapping is selected from a 
plurality of interpretation mappings. 

3. The method of claim U wherein the input data comprises handwriting. 

4. The method of claim 1» wherein the iiq)ut data comprises voice data. 

5. The metibiod of claim 1 » wherein the context data comprises data entry form 
element data. 

6. The method of claim 1, wherein the context data comprises hierarchical 
information. 

7. An input recognition system comprising: 

a context module configured to receive context input and configured to provide 
context data; 

an input capture module configured to receive input data and configured to provide 
digitized input data; and 

a recognition module coupled to the context module and coupled to the input capture 
module, the recognition module configured to receive die digitized input 
data, the recognition module configured to interpret the digitized input data 
utilizing an interpretation mapping associated wiOi the context data. 



8. The input recognition system of claun 7, wherein the interpretation mapping is 
selected from a plurality of interpretation mappings. 
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9. The input lecognition system of claim 7, wherein the input data comprises 
handwriting. 

10. The input recognition system of claim 7, wherein the input data comprises voice 

data. 

1 1 . The input recognition system of claim 7, further comprising: 
at least one additional recognition module; and 

a router module configured to utilize the context data to selectively send digitized 
input data to one selected recognition module. 

12. The input recognition system of claim 7, further comprising: 
at least one additional recognition module; and 

a multiplexor configured to utilize the context data to select symbolic output firom 
one selected recognition module. 

13. The input recognition system of claim 7, further comprising a feedback module 
configured to receive symbolic data associated with the interpretation of the digitized input 
data and configured to receive the digitized input data, the feedback module configured to 
receive user input and configured to produce feedback data. 

14. Hie ixsput recognition system of claim 7, wherein the context data comprises 
user data. 

15. The input recognition system of claim 7, whereia the context data conq)rises 
medical data. 

16. The input recognition system of claim 7, wherein the context data comprises 
template based data. 

17. The input recognition system of claim 7, wherein the context data comprises 
hierarchical data. 



18. A medical system comprising: 
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at least one input capture module configured to captuxe input data and configured to 

provide digitized input data; 
a context module configured to receive medical workflow data and configured to 

provide context data; 
a plurality of interpretation mappings, the context data associated with at least one 

interpretation mapping of the plurality of interpretation mappings; and 
a recognition module configured to generate symbolic data from the digitized input 

data utilizing the at least one mapping associated with the context data. 

19. The medical system of claim 18, wherein the context data comprises a template 
location. 

20. The medical system of claim 18, wherein the context data comprises patient 

data. 
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